Skip to content

SRE-3534 ci: adapt CI to use el9.7 for Build and Unit tests#17376

Closed
grom72 wants to merge 60 commits intomasterfrom
grom72/SRE-3522
Closed

SRE-3534 ci: adapt CI to use el9.7 for Build and Unit tests#17376
grom72 wants to merge 60 commits intomasterfrom
grom72/SRE-3522

Conversation

@grom72
Copy link
Contributor

@grom72 grom72 commented Jan 14, 2026 •
edited
Loading

This PR upgrades the majority of the CI test stages so that they are run on EL9.7.

  • Build on 9.7
  • Unit Test on EL 9.7
  • Unit Test bdev on EL 9.7
  • NLT on EL 9.7 - several tests failed - separate PR
  • Unit Test with memcheck on EL 9.7
  • Unit Test bdev with memcheck on EL 9.7 - an issue addressed by SRE-3578 test: suppress OPENSSL_init_crypto() leak #17513 - separate PR
  • Functional on EL 9.7
  • Fault injection testing on EL 9.7 - all new reported issues are because of fault injection mechanism - separate PR
  • Test RPMs on EL 9.6
  • Functional Hardware
    • Functional Hardware Medium
    • Functional Hardware Medium MD on SSD
    • Functional Hardware Medium VMD
    • Functional Hardware Medium Verbs Provider
    • Functional Hardware Medium Verbs Provider MD on SSD
    • Functional Hardware Medium UCX Provider
    • Functional Hardware Large
    • Functional Hardware Large MD on SSD

Requires:

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@grom72
SRE-3522 ci: test CI with el9.7
900350b 
Cancel-prev-build: false
Priority: 2

Skip-build-leap15-gcc: true

Allow-unstable-test: true

Skip-func-hw-test-medium: false

Skip-func-hw-test-medium-md-on-ssd: false

Skip-func-hw-test-large: false
Skip-func-hw-test-large-md-on-ssd: false

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@github-actions
Copy link

github-actions bot commented Jan 14, 2026 •
edited
Loading

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/SRE-3534

grom72 added 2 commits January 15, 2026 15:38
@grom72
Trigger FT skip NLT
8590393 
Priority: 2

Skip-unit-tests:true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Force 9.7 via Jenkins file
1750cc8 
Cancel-prev-build: false
Priority: 2

Allow-unstable-test: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 15, 2026

Test stage Unit Test bdev with memcheck on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/3/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 15, 2026

Test stage Unit Test on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/3/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 15, 2026

Test stage Unit Test bdev on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/3/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 15, 2026

Test stage Unit Test with memcheck on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/3/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 15, 2026

Test stage NLT on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/3/display/redirect

@grom72
Switch to EL9
0cabf7b 
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 15, 2026

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17376/2/execution/node/738/log

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 15, 2026

Test stage Functional on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17376/2/execution/node/748/log

@grom72
Use DISTRO provided by pipeline-lib
eded0e1 
Cancel-prev-build: false
Priority: 2

Skip-unit-tests:true

Allow-unstable-test: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 15, 2026

Test stage Unit Test with memcheck on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/4/testReport/

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 15, 2026

Test stage NLT on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/4/testReport/

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 15, 2026

Test stage Unit Test bdev with memcheck on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/4/testReport/

grom72 added 3 commits January 16, 2026 07:49
@grom72
Merge remote-tracking branch 'origin/master' into grom72/SRE-3522
afd3f01 
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Trigger HW FT on el9.7
fea5137 
Cancel-prev-build: false
Priority: 2

Skip-build-leap15-gcc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: false
Skip-unit-test-memcheck: true

Allow-unstable-test: true

Skip-func-vm: true

Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-func-hw-test-medium-vmd: false
Skip-func-hw-test-medium-verbs-provider: false
Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false
Skip-func-hw-test-medium-ucx-provider: false
Skip-func-hw-test-large: false
Skip-func-hw-test-large-md-on-ssd: false

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Trigger UT and FT on el9
f02ea77 
Cancel-prev-build: false
Priority: 2

Skip-build-leap15-gcc: true

Allow-unstable-test: true

Skip-func-vm: true

Skip-func-hw-test: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 16, 2026

Test stage Unit Test with memcheck on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/8/testReport/

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 16, 2026

Test stage NLT on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/8/testReport/

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 16, 2026

Test stage Unit Test bdev with memcheck on EL 9.7 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17376/8/testReport/

@grom72
Trigger HW FT only
a0e4556 
Cancel-prev-build: false
Priority: 2

Skip-build-leap15-gcc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: true
Skip-unit-test-memcheck: true

Skip-func-vm: true
Skip-func-test-vm-valgrind: true
Skip-func-test-el8: true
Skip-func-test-el9: true
Skip-func-test-leap15: true
Skip-fault-injection-test: true
Skip-test-el-9.6-rpms: false
Skip-test-leap-15-rpms: true

Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-func-hw-test-medium-vmd: false
Skip-func-hw-test-medium-verbs-provider: false
Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false
Skip-func-hw-test-medium-ucx-provider: false
Skip-func-hw-test-large: false
Skip-func-hw-test-large-md-on-ssd: false

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72 grom72 mentioned this pull request Jan 16, 2026
Merged
@grom72
Trigger only HW FT
3f8c7c2 
Cancel-prev-build: false
Priority: 2

Skip-build-leap15-gcc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: false
Skip-unit-test-memcheck: true

Allow-unstable-test: true

Skip-func-vm: true
Skip-func-test-vm-valgrind: true
Skip-func-test-el8: true
Skip-func-test-el9: true
Skip-func-test-leap15: true
Skip-fault-injection-test: true
Skip-test-el-9.6-rpms: true
Skip-test-leap-15-rpms: true

Skip-func-hw-test: false
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-func-hw-test-medium-vmd: false
Skip-func-hw-test-medium-verbs-provider: false
Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false
Skip-func-hw-test-medium-ucx-provider: false
Skip-func-hw-test-large: false
Skip-func-hw-test-large-md-on-ssd: false

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@daosbuild3
Copy link
Collaborator

daosbuild3 commented Jan 16, 2026

Test stage Functional on EL 9.7 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/10/display/redirect

@daosbuild3
Copy link
Collaborator

daosbuild3 commented Feb 26, 2026 •
edited by grom72
Loading

Test stage Test RPMs on EL 9.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17376/49/display/redirect

This build was only to confirm that the Fault Injection on EL 8.8 stage works properly.
Test RPMs on EL 9.6 failed because the EL9 build was disabled intentionally (Skip-build-el9-gcc: true).

osalyk
osalyk previously approved these changes Feb 26, 2026
View reviewed changes
janekmi
janekmi previously approved these changes Feb 26, 2026
View reviewed changes
@grom72 grom72 requested review from a team and daltonbohning February 26, 2026 11:04
@daltonbohning
Copy link
Contributor

daltonbohning commented Feb 26, 2026

Why is Test RPMs on EL 9.6 failing? That seems concerning since this PR updates the OS version. Also, why is that 9.6 instead of 9.7?

@grom72
Copy link
Contributor Author

grom72 commented Feb 26, 2026

Why is Test RPMs on EL 9.6 failing? That seems concerning since this PR updates the OS version. Also, why is that 9.6 instead of 9.7?

Test RPMs on EL 9.6 failed in last build, because I have disabled all things that have been already tested before as I have described in #17376 (comment)

This test is to confirm that RPM built on version N can be installed on version N-1

@daltonbohning
Copy link
Contributor

daltonbohning commented Feb 27, 2026

Since the stage name changes require updates to the merge requirements, I would like to land this next week so I can handle the requirements changes. Similar for #17176

grom72 added 4 commits March 2, 2026 11:56
@grom72
Merge remote-tracking branch 'origin/grom72/SRE-3522' into grom72/SRE…
3230306 
…-3534-final-tuning

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Merge remote-tracking branch 'origin/grom72/SRE-3534-NLT' into grom72…
03ad78e 
…/SRE-3534-final-tuning

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Merge remote-tracking branch 'origin/osalyk/SRE-3459' into grom72/SRE…
63c90c4 
…-3534-final-tuning

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Draft
51fc906 
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@daltonbohning
Copy link
Contributor

daltonbohning commented Mar 2, 2026

These three PRs conflict and overlap with each other and IMO should either be combined or written to be distinct units of work:

@daltonbohning daltonbohning removed the request for review from a team March 2, 2026 16:31
@daltonbohning daltonbohning mentioned this pull request Mar 2, 2026
Closed
6 tasks
grom72 added 7 commits March 3, 2026 07:47
@grom72
SRE-3534 ci: adapt CI to use el9.7 for Build and UT
3128349 
All EL related stages are updated to use el9.7 as default:
Build EL
Unit tests
NLT

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Cancel-prev-build: false
Priority: 2
Allow-unstable-test: true
@grom72
Trigger Functional Tests with new parameters values
f3935bb 
By default Functional EL 9 should be executed

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Cancel-prev-build: false
Priority: 2

Skip-python-bandit: true

Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: true
Skip-unit-test-memcheck: true

Allow-unstable-test: true

Skip-func-hw-test: true
@grom72
SRE-3534 ci: test Functional Test on EL9
cfb7f11 
Thest that functional tests on EL9 can be triggered without any pragmas

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Cancel-prev-build: false
Priority: 2

Skip-python-bandit: true

Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: true
Skip-unit-test-memcheck: true

Allow-unstable-test: true

Skip-fault-injection-test: true
Skip-test-el-8.6-rpms: true
Skip-test-leap-15-rpms: true

Skip-func-hw-test: true
Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
Fix Fault injection logs location
ee14072 
Fix place for fault injection logs location

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Cancel-prev-build: false
Priority: 2

Skip-python-bandit: true

Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: true
Skip-unit-test-memcheck: true

Skip-func-test-el8: true
Skip-func-test-el9: true
Skip-test-el-8.6-rpms: true
Skip-test-leap-15-rpms: true
@grom72
For pipeline-lib test purpose only
f106467 
To test that default EL9 FT works.

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Cancel-prev-build: false
Priority: 2
Test-tag: test_self_test

Skip-python-bandit: true

Skip-build-leap15-gcc: true
Skip-build-leap15-icc: true

Skip-unit-tests:true
Skip-unit-test: true
Skip-NLT: true
Skip-unit-test-memcheck: true

Allow-unstable-test: true

Skip-fault-injection-test: true
Skip-test-el-8.6-rpms: true
Skip-test-leap-15-rpms: true

Skip-func-hw-test: true
@grom72
Revert "For pipeline-lib test purpose only"
e06e3b1 
This reverts commit f106467.

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>
@grom72
CI_MORE_FUNCTIONAL_PR_TESTS is obsolete
e603a7b 
CI_MORE_FUNCTIONAL_PR_TESTS is no longer needed to trigger FT on
el8,leap15.
Only CI_FUNCTIONAL_*_TEST parameters are used to controll which tests
should be run.

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Cancel-prev-build: false
Priority: 2

Skip-func-hw-test: true
@grom72 grom72 dismissed stale reviews from janekmi and osalyk via e603a7b March 4, 2026 12:09
@grom72 grom72 requested review from janekmi and osalyk March 4, 2026 12:12
@grom72
Copy link
Contributor Author

grom72 commented Mar 4, 2026

Merge into #17632

@grom72 grom72 closed this Mar 4, 2026
@grom72 grom72 deleted the grom72/SRE-3522 branch March 4, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@janekmi janekmi janekmi approved these changes

@phender phender Awaiting requested review from phender

@JohnMalmberg JohnMalmberg Awaiting requested review from JohnMalmberg

@ryon-jensen ryon-jensen Awaiting requested review from ryon-jensen

@daltonbohning daltonbohning Awaiting requested review from daltonbohning

@osalyk osalyk Awaiting requested review from osalyk

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@grom72 @daosbuild3 @daltonbohning @janekmi @JohnMalmberg @ryon-jensen @osalyk